MusiKeys: Exploring Haptic-to-Auditory Sensory Substitution to Improve Mid-Air Text-Entry.

Physical QWERTY keyboards are the current standard for performing precision text-entry with extended reality devices. Ideally, there would exist a comparable, self-contained solution that works anywhere, without requiring external keyboards. Unfortunately, when physical keyboards are recreated virtually, we currently lose critical haptic feedback information from the sense of touch, which impedes typing. In this paper, we introduce the MusiKeys Technique, which uses auditory feedback in virtual reality to communicate missing haptic feedback information typists normally receive when using a physical keyboard. To examine this concept, we conducted a user study with 24 participants which encompassed four mid-air virtual keyboards augmented with increasing amounts of feedback information, along with a fifth physical keyboard for reference. Results suggest that providing clicking feedback on key-press and key-release improves typing performance compared to not providing auditory feedback, which is consistent with the literature. We also found that audio can serve as a substitute for information contained in haptic feedback, in that users can accurately perceive the presented information. However, under our specific study conditions, this awareness of the feedback information did not yield significant differences in typing performance. Our results suggest this kind of feedback replacement can be perceived by users but needs more research to tune and improve the specific techniques.

Exploring Bimanual Haptic Feedback for Spatial Search in Virtual Reality.

Spatial search tasks are common and crucial in many Virtual Reality (VR) applications. Traditional methods to enhance the performance of spatial search often employ sensory cues such as visual, auditory, or haptic feedback. However, the design and use of bimanual haptic feedback with two VR controllers for spatial search in VR remains largely unexplored. In this work, we explored bimanual haptic feedback with various combinations of haptic properties, where four types of bimanual haptic feedback were designed, for spatial search tasks in VR. Two experiments were designed to evaluate the effectiveness of bimanual haptic feedback on spatial direction guidance and search in VR. The results from the first experiment reveal that our proposed bimanual haptic schemes significantly enhanced the recognition of spatial directions in terms of accuracy and speed compared to spatial audio feedback. The second experiment's findings suggest that the performance of bimanual haptic feedback was comparable to or even better than the visual arrow, especially in reducing the angle of head movement and enhancing searching targets behind the participants, which was supported by subjective feedback as well. Based on these findings, we have derived a set of design recommendations for spatial search using bimanual haptic feedback in VR.

Child development and the role of visual experience in the use of spatial and non-spatial features in haptic object perception.

Previous work has suggested a different developmental timeline and role of visual experience for the use of spatial and non-spatial features in haptic object recognition. To investigate this conjecture, we used a haptic ambiguous odd-one-out task in which one object needed to be selected as being different from two other objects. The odd-one-out could be selected based on four characteristics: size, shape (spatial), texture, and weight (non-spatial). We tested sighted children from 4 to 12 years of age; congenitally blind, late blind, and adult participants with low vision; and normally sighted adults. Given the protracted developmental time course for spatial perception, we expected a shift from a preference for non-spatial features toward spatial features during typical development. Due to the dominant influence of vision for spatial perception, we expected congenitally blind adults to show a similar preference for non-spatial features as the youngest children. The results confirmed our first hypothesis; the 4-year-olds demonstrated a lower dominance for spatial features for object classification compared with older children and sighted adults. In contrast, our second hypothesis was not confirmed; congenitally blind adults' preferred categorization criteria were indistinguishable from those of sighted controls. These findings suggest an early development, but late maturation, of spatial processing in haptic object recognition independent of visual experience.

Reconsidering Luria's speech mediation: Verbalization and haptic picture identification in children with congenital total blindness.

Current accounts of behavioral and neurocognitive correlates of plasticity in blindness are just beginning to incorporate the role of speech and verbal production. We assessed Vygotsky/Luria's speech mediation hypothesis, according to which speech activity can become a mediating tool for perception of complex stimuli, specifically, for encoding tactual/haptic spatial patterns which convey pictorial information (haptic pictures). We compared verbalization in congenitally totally blind (CTB) and age-matched sighted but visually impaired (VI) children during a haptic picture naming task which included two repeated, test-retest, identifications. The children were instructed to explore 10 haptic schematic pictures of objects (e.g., cup) and body parts (e.g., face) and provide (without experimenter's feedback) their typical name. Children's explorations and verbalizations were videorecorded and transcribed into audio segments. Using the Computerized Analysis of Language (CLAN) program, we extracted several measurements from the observed verbalizations, including number of utterances and words, utterance/word duration, and exploration time. Using the Word2Vec natural language processing technique we operationalized semantic content from the relative distances between the names provided. Furthermore, we conducted an observational content analysis in which three judges categorized verbalizations according to a rating scale assessing verbalization content. Results consistently indicated across all measures that the CTB children were faster and semantically more precise than their VI counterparts in the first identification test, however, the VI children reached the same level of precision and speed as the CTB children at retest. Overall, the task was harder for the VI group. Consistent with current neuroscience literature, the prominent role of speech in CTB and VI children's data suggests that an underlying cross-modal involvement of integrated brain networks, notably associated with Broca's network, likely also influenced by Braille, could play a key role in compensatory plasticity via the mediational mechanism postulated by Luria.

[Interactive Virtual Simulation with Haptics for Neurosurgery].

We established a unique pre-surgical simulation method by applying interactive virtual simulation(IVS)using multi-fusion three-dimensional imaging data, presenting high-quality visualization of microsurgical anatomies. Our IVS provided a realistic environment for imitating surgical manipulations, such as dissecting bones, retracting brain tissues, and removing tumors, with tactile and kinesthetic sensations delivered through a specific haptic device. The great advantage of our IVS was in deciding the most appropriate craniotomy and bone resection to create the optimal surgical window and obtain the best working space with a thorough understanding of the lesion-bone relationship. Particularly for skull-base tumors, tailoring the procedures to individual patients for craniotomy and bone resection was sufficiently achieved using our IVS. In cases of large skull base meningiomas, our IVS was also helpful preoperatively regarding tumors, as several compartments were achievable in every potentially usable surgical direction. Additionally, the non-risky realistic microsurgical environments of the IVS provided improvement in the microsurgical senses and skills of young trainees through the repetition of surgical tasks. Finally, our presurgical IVS simulation method provided a realistic environment for practicing microsurgical procedures virtually and enabled us to ascertain the complex microsurgical anatomy, determine optimal surgical strategies, and efficiently educate neurosurgical trainees.

Visuo-Haptic VR and AR Guidance for Dental Nerve Block Education.

The inferior alveolar nerve block (IANB) is a dental anesthetic injection that is critical to the performance of many dental procedures. Dental students typically learn to administer an IANB through videos and practice on silicone molds and, in many dental schools, on other students. This causes significant stress for both the students and their early patients. To reduce discomfort and improve clinical outcomes, we created an anatomically informed virtual reality headset-based educational system for the IANB. It combines a layered 3D anatomical model, dynamic visual guidance for syringe position and orientation, and active force feedback to emulate syringe interaction with tissue. A companion mobile augmented reality application allows students to step through a visualization of the procedure on a phone or tablet. We conducted a user study to determine the advantages of preclinical training with our IANB simulator. We found that in comparison to dental students who were exposed only to traditional supplementary study materials, dental students who used our IANB simulator were more confident administering their first clinical injections, had less need for syringe readjustments, and had greater success in numbing patients.

Dislocation force of scleral flange-fixated intraocular lens haptics.

PURPOSE: To measure the dislocation forces in relation to haptic material, flange size and needle used. SETTING: Hanusch Hospital, Vienna, Austria. DESIGN: Laboratory Investigation. METHODS, MAIN OUTCOME MEASURES: 30 G (gauge) thin wall and 27 G standard needles were used for a 2 mm tangential scleral tunnel in combination with different PVDF (polyvinylidene fluoride) and PMMA (polymethylmethacrylate haptics). Flanges were created by heating 1 mm of the haptic end, non-forceps assisted in PVDF and forceps assisted in PMMA haptics. The dislocation force was measured in non-preserved cadaver sclera using a tensiometer device. RESULTS: PVDF flanges achieved were of a mushroom-like shape and PMMA flanges were of a conic shape. For 30 G needle tunnels the dislocation forces for PVDF and PMMA haptic flanges were 1.58 ± 0.68 N (n = 10) and 0.70 ± 0.14 N (n = 9) (p = 0.003) respectively. For 27 G needle tunnels the dislocation forces for PVDF and PMMA haptic flanges were 0.31 ± 0.35 N (n = 3) and 0.0 N (n = 4), respectively. The flange size correlated with the occurring dislocation force in experiments with 30 G needle tunnels (r = 0.92), when flanges were bigger than 384 micrometres. CONCLUSIONS: The highest dislocation forces were found for PVDF haptic flanges and their characteristic mushroom-like shape for 30 G thin wall needle scleral tunnels. Forceps assisted flange creation in PMMA haptics did not compensate the disadvantage of PMMA haptics with their characteristic conic shape flange.

Enhancement of sense of ownership using virtual and haptic feedback.

Accomplishing motor function requires multimodal information, such as visual and haptic feedback, which induces a sense of ownership (SoO) over one's own body part. In this study, we developed a visual-haptic human machine interface that combines three different types of feedback (visual, haptic, and kinesthetic) in the context of passive hand-grasping motion and aimed to generate SoO over a virtual hand. We tested two conditions, both conditions the three set of feedback were synchronous, the first condition was in-phase, and the second condition was in antiphase. In both conditions, we utilized passive visual feedback (pre-recorded video of a real hand displayed), haptic feedback (balloon inflated and deflated), and kinesthetic feedback (finger movement following the balloon curvature). To quantify the SoO, the participants' reaction time was measured in response to a sense of threat. We found that most participants had a shorter reaction time under anti-phase condition, indicating that synchronous anti-phase of the multimodal system was better than in-phase condition for inducing a SoO of the virtual hand. We conclude that stronger haptic feedback has a key role in the SoO in accordance with visual information. Because the virtual hand is closing and the high pressure from the balloon against the hand creates the sensation of grasping and closing the hand, it appeared as though the person was closing his/her hand at the perceptual level.

Prior visual experience increases children's use of effective haptic exploration strategies in audio-tactile sound-shape correspondences.

Sound-shape correspondence refers to the preferential mapping of information across the senses, such as associating a nonsense word like bouba with rounded abstract shapes and kiki with spiky abstract shapes. Here we focused on audio-tactile (AT) sound-shape correspondences between nonsense words and abstract shapes that are felt but not seen. Despite previous research indicating a role for visual experience in establishing AT associations, it remains unclear how visual experience facilitates AT correspondences. Here we investigated one hypothesis: seeing the abstract shapes improve haptic exploration by (a) increasing effective haptic strategies and/or (b) decreasing ineffective haptic strategies. We analyzed five haptic strategies in video-recordings of 6- to 8-year-old children obtained in a previous study. We found the dominant strategy used to explore shapes differed based on visual experience. Effective strategies, which provide information about shape, were dominant in participants with prior visual experience, whereas ineffective strategies, which do not provide information about shape, were dominant in participants without prior visual experience. With prior visual experience, poking-an effective and efficient strategy-was dominant, whereas without prior visual experience, uncategorizable and ineffective strategies were dominant. These findings suggest that prior visual experience of abstract shapes in 6- to 8-year-olds can increase the effectiveness and efficiency of haptic exploration, potentially explaining why prior visual experience can increase the strength of AT sound-shape correspondences.

Developmental changes in the visual, haptic, and bimodal perception of geometric angles.

Geometrical knowledge is typically taught to children through a combination of vision and repetitive drawing (i.e. haptics), yet our understanding of how different spatial senses contribute to geometric perception during childhood is poor. Studies of line orientation suggest a dominant role of vision affecting the calibration of haptics during development; however, the associated multisensory interactions underpinning angle perception are unknown. Here we examined visual, haptic, and bimodal perception of angles across three age groups of children: 6 to 8 years, 8 to 10 years, and 10 to 12 years, with age categories also representing their class (grade) in primary school. All participants first learned an angular shape, presented dynamically, in one of three sensory tracing conditions: visual only, haptic only, or bimodal exploration. At test, which was visual only, participants selected a target angle from four possible alternatives with distractor angle sizes varying relative to the target angle size. We found a clear improvement in accuracy of angle perception with development for all learning modalities. Angle perception in the youngest group was equally poor (but above chance) for all modalities; however, for the two older child groups, visual learning was better than haptics. Haptic perception did not improve to the level of vision with age (even in a comparison adult group), and we found no specific benefit for bimodal learning over visual learning in any age group, including adults. Our results support a developmental increment in both spatial accuracy and precision in all modalities, which was greater in vision than in haptics, and are consistent with previous accounts of cross-sensory calibration in the perception of geometric forms.

Effect of Decentration, Rotation, and Tilt on Objective Optical Quality of Plate Haptic Toric Intraocular Lenses in the Early Postoperative Period.

Purpose: This study aimed to determine the influence of decentration, rotation, and tilt on objective optical quality of plate haptic toric intraocular lenses (tIOLs). Methods: The area ratio of modulation transfer function (MTF), strehl ratio of point spread function (PSF), and higher order aberrations (HOAs) for 3 mm and 5 mm pupil diameter (PD) were evaluated at postoperative 1 month. The retroillumination images pictured by OPD-scan III were used to quantify the degree of decentration and rotation, whereas the tIOL tilt was directly obtained by the tilt aberration. Patients were separated into two subgroups based on tIOL misalignment cutoff values. Results: There were 29 eyes (24 patients) in the study. The decentration of more than 0.25 mm did not substantially differ from those less than or equal to 0.25 mm. PSF of 3 mm PD and MTF, intraocular HOAs, and trefoil aberration for 3 mm and 5 mm PD significantly deteriorated with a rotation of more than 3 degrees, whereas only intraocular HOAs for 5 mm PD and coma for 3 mm and 5 mm PD were significantly severe with a tilt of more than 0.1 µm and 0.25 µm in corresponding PD. Furthermore, tIOL rotation and tilt were highly correlated with intraocular trefoil aberration and coma, respectively. Conclusions: The decentration of the monofocal bitoric IOLs is more tolerant to optical quality degradation after 1 month of surgery but more sensitive to intraocular trefoil aberration caused by rotation and coma aberration induced by tilt. Translational Relevance: As far as we know, this is the first study to investigate the relationship between the plate haptic bitoric IOL misalignment and objective optical quality measured by OPD-scan III in the real world, which may provide reference information for IOL selection to improve surgical outcomes.

Multimodal Haptic Feedback for Virtual Collisions Combining Vibrotactile and Electrical Muscle Stimulation.

In this paper, we explore the effects of multimodal haptic feedback combining vibrotactile and electrical muscle stimulation (EMS) on expressing virtual collisions. We first present a wearable multimodal haptic device capable of generating both mechanical vibration and EMS stimuli. The two types of haptic stimulus are combined into a haptic rendering method that conveys improved virtual collision sensations. This multimodal rendering method highlights the strengths of each modality while compensating for mutual weaknesses. The multimodal rendering method is compared in subjective quality with two unimodal methods (vibration only and EMS only) by a user study. Experimental results demonstrate that our multimodal feedback method can elicit more realistic, enjoyable, expressive, and preferable user experiences.

Horizontal Plane Haptic Redirection: Realizing Haptic Feedback for the Virtual Inclined Plane in VR.

Some interactions in virtual environments need to be operated on inclined planes. If a real inclined plane can be found in the real environment that corresponds exactly to the angle of the virtual inclined plane to provide haptic feedback, the user's immersion can be enhanced. However, it is not easy to find such a real inclined plane in the real environment. We proposed a horizontal plane haptic redirection scheme, where users interacting with virtual inclined planes in virtual environments can obtain haptic feelings by using real horizontal planes that are easily available in the real world for redirection mapping. We also designed an integrated solution to locate the real horizontal plane and for haptic redirection based on the Vive Pro headset. Then we measured the angle and size thresholds for horizontal plane haptic redirection as 20° and 88%, respectively, through a user study. Through experiments, we also found that when the degree of redirection exceeded the threshold, the user's operation efficiency would be significantly reduced. In addition, we compared the horizontal plane haptic redirection scheme with the scheme without redirection and the scheme without haptic feedback to demonstrate the validity and necessity of the redirection scheme proposed in this paper.

Robust Surface Recognition With the Maximum Mean Discrepancy: Degrading Haptic-Auditory Signals Through Bandwidth and Noise.

Sliding a tool across a surface generates rich sensations that can be analyzed to recognize what is being touched. However, the optimal configuration for capturing these signals is yet unclear. To bridge this gap, we consider haptic-auditory data as a human explores surfaces with different steel tools, including accelerations of the tool and finger, force and torque applied to the surface, and contact sounds. Our classification pipeline uses the maximum mean discrepancy (MMD) to quantify differences in data distributions in a high-dimensional space for inference. With recordings from three hemispherical tool diameters and ten diverse surfaces, we conducted two degradation studies by decreasing sensing bandwidth and increasing added noise. We evaluate the haptic-auditory recognition performance achieved with the MMD to compare newly gathered data to each surface in our known library. The results indicate that acceleration signals alone have great potential for high-accuracy surface recognition and are robust against noise contamination. The optimal accelerometer bandwidth exceeds 1000 Hz, suggesting that useful vibrotactile information extends beyond human perception range. Finally, smaller tool tips generate contact vibrations with better noise robustness. The provided sensing guidelines may enable superhuman performance in portable surface recognition, which could benefit quality control, material documentation, and robotics.

Comparative analysis of outcomes of two (popular) techniques of haptic exteriorization in scleral fixation intra-ocular lens surgery - A retrospective study.

PURPOSE: To compare the visual outcomes and complication rates between the extra-ocular needle-guided haptic insertion technique (XNIT) and the conventional handshake (HS) technique of scleral fixation intra-ocular lens (SFIOL). METHODS: In this retrospective study, we retrieved data of those patients who had undergone SFIOL surgery from January 2018 to May 2022 at our institute for aphakia following either a complicated cataract surgery or an ocular trauma and had a minimum follow-up of 3 months. RESULTS: Of the 156 eyes, the HS technique was done in 80 eyes and the remaining 76 eyes with XNIT. At 3 months follow-up visit, there was no significant difference in the median best corrected visual acuity (BCVA) ( P = 0.988) and uncorrected visual acuity (UCVA) ( P = 0.765) between the two techniques. There was no statistically significant difference between pre-operative median BCVA and post-operative UCVA in XNIT ( P = 0.961) and the HS technique ( P = 0.831) at 3 months follow-up visit. The complication rates between the two techniques were minimal and comparable. The most common post-operative complication was corneal edema. The incidence of cystoid macular edema was slightly more in the XNIT group but not statistically significant ( P = 0.05). Two patients in the HS group developed retinal detachment, which settled after repeat surgery. CONCLUSION: The newer XNIT technique was found to be as safe and effective as compared to the conventional HS technique.

Modality-Independent Effect of Gravity in Shaping the Internal Representation of 3D Space for Visual and Haptic Object Perception.

Visual and haptic perceptions of 3D shape are plagued by distortions, which are influenced by nonvisual factors, such as gravitational vestibular signals. Whether gravity acts directly on the visual or haptic systems or at a higher, modality-independent level of information processing remains unknown. To test these hypotheses, we examined visual and haptic 3D shape perception by asking male and female human subjects to perform a "squaring" task in upright and supine postures and in microgravity. Subjects adjusted one edge of a 3D object to match the length of another in each of the three canonical reference planes, and we recorded the matching errors to obtain a characterization of the perceived 3D shape. The results show opposing, body-centered patterns of errors for visual and haptic modalities, whose amplitudes are negatively correlated, suggesting that they arise in distinct, modality-specific representations that are nevertheless linked at some level. On the other hand, weightlessness significantly modulated both visual and haptic perceptual distortions in the same way, indicating a common, modality-independent origin for gravity's effects. Overall, our findings show a link between modality-specific visual and haptic perceptual distortions and demonstrate a role of gravity-related signals on a modality-independent internal representation of the body and peripersonal 3D space used to interpret incoming sensory inputs.

Haptic Interactions Subject to Variable Latency.

Model-Mediated Teleoperation (MMT) between a haptic device and a remote or virtual environment uses a local model of the environment to compensate for latency of communication. MMT is often case-specific, and requires underlying latency distributions to be known. We propose a novel approach - which we refer to as the DelayRIM - which uses the time-stepping aspect of a Reduced Interface Model for the environment to render an up-to-date force to the haptic device from the delayed information. RIM is applicable to any physical or virtual system, and the DelayRIM itself makes no underlying assumption about the latency distribution. We show that for realistic variable delays, the DelayRIM improves transparency compared to other methods for a virtual drone bilateral teleoperation scenario.

Telemetry-Based Haptic Rendering for Racing Game Experience Improvement.

Many recent games, such as racing and flight games, open their game telemetry data to users by storing them in the local memory. Such telemetry data can provide useful information for haptic rendering, and this advantage has been exploited by the industry. This approach applies to any applications that export telemetry data in run time. The haptic rendering module operates as a separate process that accesses the telemetry data in parallel with the application. It is simple, efficient, and modular while retaining the application intact. We examine the approach's viability for user experience improvement by developing three telemetry-based haptic rendering algorithms for car racing games. They express the car engine response, collisions with external objects, and the road surface texture, respectively. Building a haptics-enabled driving platform, we conducted a user study comparing gaming experiences between our telemetry-based algorithms and conventional sound-to-tactile conversion algorithms. The results showed that the telemetry-based effects elicited better experiences than the sound-based effects.

Enhancing User Performance by Adaptively Changing Haptic Feedback Cues in a Fitts's Law Task.

Enhancing human user performance in some complex task is an important research question in many domains from skilled manufacturing to rehabilitation and surgical training. Many examples in the literature explore the effects of both haptic assistance or guidance to complete a task, as well as haptic hindrance to temporarily increase task difficulty for the ultimate goal of faster learning. Studies also suggest adaptively changing guidance based on expertise may be most effective. However, to our knowledge, there has not yet been a conclusive study evaluating these enhancement modes in a systematic experiment. In this article, we evaluate learning outcomes for 24 human subjects in a randomized control trial performing a Fitt's law reaching task under various haptic feedback conditions including: no haptics, assistive haptics, resistive haptics, and adaptively changing haptics tied to current performance measures. Subjects each performed 400 trials total and this paper reports results for 40 pre-test and 40 post-test trials. While most conditions did show improvements in performance, we found statistically significant results indicating that our adaptive haptic feedback condition leads to faster and more effective learning as evidenced by metrics of movement time, overshoot, performance index, and speed when compared to the other groups.

Relaxing Conservatism for Enhanced Impedance Range and Transparency in Haptic Interaction.

The Time Domain Passivity Approach (TDPA) has been accepted as one of least conservative tools for designing stabilizing controllers in haptics and teleoperation, but it still suffers from conservatism because it is based on passivity. Additionally, high-frequency, immediate control actions lead to a degradation of transparency. In this paper, we propose a method to relax the conservatism of haptic interaction and enhance stable impedance range while maintaining high transparency. Based on the observation of energy exchange behavior in pressing and releasing paths in haptic interaction, we introduce an energy cycle as a completion of a pressing and releasing path. With this new concept, we compare the energies at the end of each energy cycle to estimate the energy generation and inject adaptive damping to regulate it over upcoming cycles. Because we wait a pressing-releasing cycle is completed, we allow energy to be generated, but we regulate the amount of generated energy over upcoming cycles by injecting adaptive damping. In this way, we perform low-frequency control actions on system dynamics. These in turn enable us to achieve high transparency. We show the validity of the proposed approach through several simulations and experiments, and show that it enhances the stable impedance range and transparency compared to the TDPA.